@yiyixuxu Here for serializing PipelineBlocks, we deliberately set the library, class tuple to None. But type_hint also provides this information. Do we need to have both of these in the config then? Will modular configs ever have the class library tuple be non null values?

I don't think we need to set them to None, they should just be the actual library/class

but, yes that's a good question, let's try to find a solution here:

we do not need these two fields technically, I left it there because that's I want to have an intuitive way to be able to tell if a components are loaded or not, in DiffusionPipeline these tuples will be null values if not loaded so I did same with modular.

maybe we can keep these 2 fields (or something else that can indicate whehter they are loaded or not) in config dict ( which gets print out when you do print(modular_pipeline) ), but remove them when saving to the modular_model_index.json?

let me know what you think!

ohh actually, do you include the components info in the config.json for pipeline blocks too?
I was thinking maybe we don't need this, because pipeline blocks are just definations, so it does not need to include the loading related info - these are already saved in the pipeline, https://huggingface.co/YiYiXu/modular-diffdiff-0704

updates ModularPipelineBlock serialization so that type hint, repo, subfolder, variant information gets saved when calling block.save_pretrained. Useful for when you have custom pipeline blocks that have components that use from_pretrained

so not sure if I understand here, I think the block's config.json is only used to create block, the auto_map points to the code defination is enough becausee you can define the default loading spec inside the custom code; user may want to change the loading spec at run time but they need to do that with the ModuarPipeline's config, not the blocks, no?

Hmm so what about custom pipeline blocks that are meant to work with a particular model type? e.g Trying to create a custom block here to work with the Florence2-large model.
https://huggingface.co/diffusers-internal-dev/florence2-image-annotator/blob/main/config.json

How should one structure this? The goal is to insert a step that automatically creates a mask for an image
for inpainting based on a text prompt.

import torch
from diffusers.modular_pipelines import ModularPipelineBlocks, SequentialPipelineBlocks
from diffusers.modular_pipelines.stable_diffusion_xl import INPAINT_BLOCKS
from diffusers.utils import load_image

# fetch the Florence2 image annotator block that will create our mask
image_annotator_block = ModularPipelineBlocks.from_pretrained("diffusers-internal-dev/florence2-image-annotator", trust_remote_code=True)

my_blocks = INPAINT_BLOCKS.copy()

# insert the annotation block before the image encoding step
my_blocks.insert("image_annotator", image_annotator_block, 1)

# Create our initial set of inpainting blocks
blocks = SequentialPipelineBlocks.from_blocks_dict(my_blocks)
pipe = blocks.init_pipeline()

If the component spec information like repo/subfolder for the model is not present in the config, the init_pipeline call won't work?

so I see you already defined here https://huggingface.co/diffusers-internal-dev/florence2-image-annotator/blob/main/block.py#L17

so once you convert the blocks into pipeline, it will show up in pipeline's config like this

>>> image_annotator_block.init_pipeline()
ModularPipeline {
  "_blocks_class_name": "Florence2ImageAnnotatorBlock",
  "_class_name": "ModularPipeline",
  "_diffusers_version": "0.35.0.dev0",
  "image_annotator": [
    null,
    null,
    {
      "repo": "microsoft/Florence-2-large",
      "revision": null,
      "subfolder": "",
      "type_hint": [
        "transformers",
        "AutoModelForCausalLM"
      ],
      "variant": null
    }
  ],
  "image_annotator_processor": [
    null,
    null,
    {
      "repo": "microsoft/Florence-2-large",
      "revision": null,
      "subfolder": "",
      "type_hint": [
        "transformers",
        "AutoProcessor"
      ],
      "variant": null
    }
  ]
}

and if you run save_pretrained() on the pipeline, you will get the modular_model_index.json contains the loading specs for the cusom model

pipe = image_annotator_block.init_pipeline()
pipe.save_pretraind(...)

e.g. this is what I got by just adding pipe.save_pretrained("YiYiXU/test_image_annotator", push_to_hub=True) to the code example you shared https://huggingface.co/YiYiXu/test_image_annotator

ideally, we also push the custom code in the same repo, so everything can be together with ModularPipeline.from_pretrained() (the custom blocks + loading configs -> you got a fully initialized custom pipeline)

and since everything is in the same folder, the author of the custom pipeline also have the option to fill out the repo/subfolder info in modular_model_index.json if they like (instead of defining it in the code)

Ah I see it is expected to load from the custom code itself. Okay, so I made changes to ModularPipelineBlocks to allow running init_pipeline lmk if those look okay.

Re: Having all components + custom code in the same repo. Assuming we load two custom blocks from two different repos, each has a block.py file defining the custom code. If we serialize the Pipeline with code into one repo, how do we figure out where to put all the components? Also how do we handle components with custom code in modular_model_index.json?

Re: (None, None, <config details>) format in modular_model_index.json and this

maybe we can keep these 2 fields (or something else that can indicate whehter they are loaded or not) in config dict ( which gets print out when you do print(modular_pipeline) ), but remove them when saving to the modular_model_index.json?

I think this could work. I can put together a quick PR to just see how it feels.

Re: Having all components + custom code in the same repo. Assuming we load two custom blocks from two different repos, each has a block.py file defining the custom code. If we serialize the Pipeline with code into one repo, how do we figure out where to put all the components? Also how do we handle components with custom code in modular_model_index.json?

For the blocks author, they can define the default component loading specs (repo/subfolder etc) when they write the pipeline blocks code, in the expected_components field. It's optional to include the repo info, but if they did that, like you did here with the florence-image-annotator block https://huggingface.co/diffusers-internal-dev/florence2-image-annotator/blob/main/block.py#L17 - these info info will stick with the block wherever it goes forever. lol

so in your question, if you load two different custom blocks from two different repos and mix them together into one pipeline, the loading specs for these custom components from both blocks should show up as default in your final pipeline. If you save the final pipeline into a repo, it will serialize the pipeline config dict into a modular_model_index.json in that repo, which will contain the default loading spec for the custom components, but the person who put together the final pipeline can edit the modular_model_index.json if they want to use something else. The end user could use a different loading specs at run time as well

how do we handle components with custom code in modular_model_index.json?

not sure I understand the question correctly, but ModularPipeline.from_pretrained() will look for custom code first, so if you save the code in the same repo, it should work
the code is here:
https://github.com/huggingface/diffusers/blame/main/src/diffusers/modular_pipelines/modular_pipeline.py#L2053

if your pipeline uses two different blocks from two different repos, I imagine your repo also need a block.py, and in that block.py you import these custom code, so I think it would also work lol but have not tested with such cases so let me know

and if you only want to host the block in your repo, (not the entire pipeline and do not want to save a modular_model_index.json) - defining the custom componenet in the code is enough,

I think your current code alreaedy work out of box very well: if the user just want to import that custom block, mix into their existing inpanting pipeline, they can pass the inpaiting repo in init_pipeline() - the final pipeline will contain the loading info for the custom component (since we defined a default) and all the loading info for other components (fetchd from the modular_model_index.json in that inpainting repo we passed)

e.g.
your code, only difference is I passed a "YiYiXu/modular-demo-auto" to init_pipeline() - "YiYiXu/modular-demo-auto" does not contain the custom component but we have a default defined so pipeline still have that info

import torch
from diffusers.modular_pipelines import ModularPipelineBlocks, SequentialPipelineBlocks
from diffusers.modular_pipelines.stable_diffusion_xl import INPAINT_BLOCKS
from diffusers.utils import load_image

# fetch the Florence2 image annotator block that will create our mask
from diffusers.modular_pipelines import PipelineBlock
image_annotator_block = ModularPipelineBlocks.from_pretrained("diffusers-internal-dev/florence2-image-annotator", trust_remote_code=True)

my_blocks = INPAINT_BLOCKS.copy()

# insert the annotation block before the image encoding step
my_blocks.insert("image_annotator", image_annotator_block, 1)

# Create our initial set of inpainting blocks
blocks = SequentialPipelineBlocks.from_blocks_dict(my_blocks)
pipe = blocks.init_pipeline("YiYiXu/modular-demo-auto")
print(pipe)

the pipeline is ready to use and knows how to load all the components

StableDiffusionXLModularPipeline {
  "_blocks_class_name": "SequentialPipelineBlocks",
  "_class_name": "StableDiffusionXLModularPipeline",
  "_diffusers_version": "0.35.0.dev0",
  "force_zeros_for_empty_prompt": true,
  "image_annotator": [
    null,
    null,
    {
      "repo": "microsoft/Florence-2-large",
      "revision": null,
      "subfolder": "",
      "type_hint": [
        "transformers",
        "AutoModelForCausalLM"
      ],
      "variant": null
    }
  ],
  "image_annotator_processor": [
    null,
    null,
    {
      "repo": "microsoft/Florence-2-large",
      "revision": null,
      "subfolder": "",
      "type_hint": [
        "transformers",
        "AutoProcessor"
      ],
      "variant": null
    }
  ],
  "requires_aesthetics_score": false,
  "scheduler": [
    null,
    null,
    {
      "repo": "SG161222/RealVisXL_V4.0",
      "revision": null,
      "subfolder": "scheduler",
      "type_hint": [
        "diffusers",
        "EulerDiscreteScheduler"
      ],
      "variant": null
    }
  ],
  "text_encoder": [
    null,
    null,
    {
      "repo": "SG161222/RealVisXL_V4.0",
      "revision": null,
      "subfolder": "text_encoder",
      "type_hint": [
        "transformers",
        "CLIPTextModel"
      ],
      "variant": null
    }
  ],
  "text_encoder_2": [
    null,
    null,
    {
      "repo": "SG161222/RealVisXL_V4.0",
      "revision": null,
      "subfolder": "text_encoder_2",
      "type_hint": [
        "transformers",
        "CLIPTextModelWithProjection"
      ],
      "variant": null
    }
  ],
  "tokenizer": [
    null,
    null,
    {
      "repo": "SG161222/RealVisXL_V4.0",
      "revision": null,
      "subfolder": "tokenizer",
      "type_hint": [
        "transformers",
        "CLIPTokenizer"
      ],
      "variant": null
    }
  ],
  "tokenizer_2": [
    null,
    null,
    {
      "repo": "SG161222/RealVisXL_V4.0",
      "revision": null,
      "subfolder": "tokenizer_2",
      "type_hint": [
        "transformers",
        "CLIPTokenizer"
      ],
      "variant": null
    }
  ],
  "unet": [
    null,
    null,
    {
      "repo": "SG161222/RealVisXL_V4.0",
      "revision": null,
      "subfolder": "unet",
      "type_hint": [
        "diffusers",
        "UNet2DConditionModel"
      ],
      "variant": null
    }
  ],
  "vae": [
    null,
    null,
    {
      "repo": "SG161222/RealVisXL_V4.0",
      "revision": null,
      "subfolder": "vae",
      "type_hint": [
        "diffusers",
        "AutoencoderKL"
      ],
      "variant": null
    }
  ]
}

why it's only PipelineBlock?

users should be able to define custom ModularPipelineBlocks too where they assmeble their own presets so that it is easier to use

in diff-diff remote code example, https://huggingface.co/YiYiXu/modular-diffdiff-0704/blob/main/config.json

maybe support all block types?

Ah. Yeah that make sense will update.